Methods and associated devices and systems for enhanced 2d and 3d vision

ABSTRACT

Methods, devices and systems are disclosed for improved depth perception in stereoscopic night vision devices. Among these are embodiments for aligning information overlays in the stereo view with associated objects, and for generating stereo information from single lenses or intensifiers. In some illustrative embodiments, a camera and position sensor are provided for at least two viewers, e.g., a pilot and a copilot, such that when a scene overlaps between viewers, the system produces a stereoptic scene, in which the users can more accurately determine a difference in depth between two or more distant objects. An illustrative binocular night vision system uses a high-resolution depth map to present binocular images to a user. In some embodiments, supplementary content can be overlaid, with an appropriate binocular disparity that is based on the depth map.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation of U.S. application Ser. No.15/663,617, filed 28 Jul. 2017, which issued as U.S. Pat. No. 10,805,600on 13 Oct. 2020, which claims priority to U.S. Provisional ApplicationNo. 62/368,846, which was filed on 29 Jul. 2016, wherein each isincorporated herein in its entirety by this reference thereto.

FIELD OF THE INVENTION

At least one embodiment of the present invention pertains to methods andassociated devices and systems for enhancing cameras and imagers. Somespecific embodiments of the present invention pertains to methods andassociated devices and systems for enhancing depth information in nightvision cameras and imagers.

BACKGROUND

The following background information may present examples of specificaspects of the prior art (e.g., without limitation, approaches, facts,or common wisdom) that, while expected to be helpful to further educatethe reader as to additional aspects of the prior art, is not to beconstrued as limiting the present invention, or any embodiments thereof,to anything stated or implied therein or inferred thereupon. It iscontemplated that many conventional night vision systems may typicallygenerate stereo images (albeit monochromatic), wherein theseconventional systems may often produce artifacts that interfere witheffective depth perception. In some other conventional imaging and/ordisplay systems, additional image information may typically be displayedin association with viewed objects wherein the additional imageinformation may appear to “jump” forward to a depth of an occludingobject, while a viewer may still view that the associated viewed objectremained at an original depth behind the occluding object. By way ofeducational background, another aspect of the prior art generally usefulto be aware of is that conventional prior art methods and systems may bemechanically complex, power-consuming intensive, and/or heavy.

SUMMARY OF THE INVENTION

Disclosed are systems, methods and devices for improved depth perceptionin stereoscopic night vision devices. Among these are techniques foraligning information overlays in the stereo view with associated objectsand generating stereo information from single lenses and/orintensifiers.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present invention are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 is a partial schematic view of an illustrative stereoptic visionsystem having separate inputs from cameras or optical devices.

FIG. 2 is a schematic view of an operating environment in which anillustrative stereoptic vision system can be used.

FIG. 3 is schematic view of camera baselines as a function of camerarotation for different viewers USR, e.g., a pilot and a copilot.

FIG. 4 is a flowchart of an illustrative method for producing astereoptic view for overlapping scenes of tracked camera positions.

FIG. 5 shows an illustrative system for augmenting stereo vision using athree-dimensional (3D) depth map.

FIG. 6 shows an illustrative method for augmenting stereo vision using athree-dimensional (3D) depth map.

FIGS. 7-10 show schematic views of system embodiments in which a depthmap can be applied to an information overlay to resolve occlusions.

FIG. 11 is a schematic view of an illustrative system for directpainting of overlay information onto a phosphor screen.

FIG. 12 is a flowchart of an illustrative method for direct painting ofoverlay information onto a phosphor screen.

FIG. 13 is a schematic view of an alternate illustrative system fordirect painting of overlay information onto a phosphor screen.

FIG. 14 is a flowchart of an alternate illustrative method for directpainting of overlay information onto a phosphor screen.

FIG. 15 is a schematic view of an illustrative system for generatingstereo information from a single lens.

FIG. 16 is a flowchart of an illustrative method for generating stereoinformation from a single lens.

FIG. 17 is a schematic view of an alternate illustrative system forgenerating stereo information from a single lens.

FIG. 18 is a flowchart of an alternate illustrative method forgenerating stereo information from a single lens.

FIG. 19 is a schematic view of an illustrative night vision device thatis configured for binocular display of the output of different opticaldevices.

FIG. 20 is a schematic view of an illustrative system for generating acomposited image signal from different optical devices, using a variablebeamsplitter.

FIG. 21 is a schematic view of an illustrative autofocus system forhead-mounted night vision goggles, which is configured to set the focusof the goggles for near vision when the user's head is tilted down.

FIG. 22 is a schematic view of an illustrative autofocus system forhead-mounted night vision goggles, which is configured to set the focusof the goggles for far vision when the user's head is looking at horizonor vertically upward.

FIG. 23 is a schematic view of illustrative optical devices foroffensive and defensive operations.

FIG. 24 is a schematic view of an illustrative apparatus for producing astereo color image from a single color channel.

DETAILED DESCRIPTION

References in this description to “an embodiment”, “one embodiment”, orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe present invention. Occurrences of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,the embodiments referred to also are not necessarily mutually exclusive.

In some exemplary embodiments of night vision systems, two imagers, onein front of each eye, may be utilized to produce stereo images. In someother exemplary embodiments, binocular night vision (NV) systems mayincorporate two image intensifier tubes, wherein an optical axis of eachimage intensifier tube may be aligned with one of the user's eyes. It isbelieved that aligning the optical axes of the image intensifier tubeswith the user's eyes optimally provides that the binocular disparity ofthe imagery presented by the image intensifier tubes matches that ofimagery (that would otherwise be) acquired by the eyes directly. Theuser is thus readily able to fuse the two presented images into a singleCyclopean image.

Furthermore, in many embodiments, augmented reality content such as, andwithout limitation, informational overlays, may provide additionalvisual data to a viewer via display by NV systems. For example, andwithout limitation, informational overlays that directly correspond tophysical objects at particular physical distances, (e.g., labelsannotating identified and tracked objects of interest within the fieldof view) should sensibly be presented with a binocular disparitycorresponding to the physical distances. Such content may be presentedto a user in one eye or in both eyes.

In another non-limiting example, informational overlays, (i.e., labels,symbols and/or other graphics corresponding to objects at particularlocation within the field of view) can be generated in each eye with abinocular disparity based on a value of a depth map at that particularlocation. It is believed that generating the overlay with a disparitymatching the depth of the corresponding object greatly aids the user'ssense of depth within the scene. The imagery acquired by imageintensifiers corresponds to real world subject matter. As the userdevelops a mental model of his or her surroundings that corresponds tothe physical world, realizing the benefits of binocular vision will leadto the contemplation of subject matter being presented across the twoeyes with a disparity matching that of the physical world.

Introduced here are methods, systems and devices that improve depthperception in stereoscopic night vision devices. Among these areembodiments for aligning information overlays in the stereo view withassociated objects, and for generating stereo information from singlelenses or intensifiers.

In certain embodiments, a camera and position sensor are provided for atleast two viewers, e.g., a pilot and a copilot, such that when a sceneoverlaps between viewers, the system produces a stereoptic scene, inwhich the users can more accurately determine a difference in depthbetween two or more distant objects.

In some embodiments, an illustrative binocular night vision system usesa high-resolution depth map to present binocular images to a user.

In some embodiments, supplementary content can be overlaid, with anappropriate binocular disparity that is based on the depth map. Forexample, supplementary information can be overlaid onto a phosphorscreen integrated with night vision goggles (NVG), such as with abeam-splitter or with a low-powered infrared laser.

Some embodiments can generate stereo information from a single lens,e.g., using filters or sensors, which can be used for operation ofremote controlled vehicles in underground/low light environments.

Other illustrative embodiments are configured to produce a stereo colorimage from a single color channel. Embodiments for automated focusing ofNVGs are also disclosed, which can be based on user movement or action.Additionally, the lenses of an NVG can be set to converge in steps, tosimulate distance.

Improved Depth Perception

Stereoscopy is a technique for creating or enhancing the illusion ofdepth in an image by means of stereopsis for binocular vision.Generally, these methods present two offset images separately to theleft and right eye of the viewer. These two-dimensional images are thencombined in the brain to give the perception of depth.

Stereoscopy is used to create 3D theatrical movies, but the sametechnique is also used in night vision goggles, thermal vision goggles,and other head mounted display devices. In these systems, the usertypically wears a helmet or glasses with two small displays, one foreach eye. The images shown on these displays are captured by twoseparate cameras or optical devices. These systems are often used byhelicopter pilots flying in low light conditions.

The distance between the two cameras is generally called the baseline.For general purpose stereo photography, where the goal is to duplicatenatural human vision and give a visual impression as close as possibleto reality, the correct baseline would be the same as the distancebetween the eyes, which is generally around 65 mm.

If a stereo picture is taken of a large, distant object such as amountain or a large building using a normal baseline, it will appear tobe flat or lacking in depth. This is in keeping with normal humanvision. To provide great depth detail in distant objects, the camerapositions can be separated by a larger distance. This will effectivelyrender the captured image as though it was seen by a giant, and thuswill enhance the depth perception of these distant objects, and reducethe apparent scale of the scene proportionately.

FIG. 1 is a partial schematic view of an illustrative stereoptic visionsystem 10 having separate inputs 24 from cameras or optical devices 12.FIG. 2 is a schematic view of an operating environment 200 in which anillustrative stereoptic vision system 10 can be used, such as withrespect to an X-axis 220 x, a Y-axis 220 y, and a Z-axis 220 z. FIG. 3is schematic view 300 of camera baselines 16 as a function of camerarotation 304 for different viewers USR, e.g., a pilot 208 a and acopilot 208 b. FIG. 4 is a flowchart of an illustrative method 400 forproducing a stereoptic view for overlapping scenes of tracked 402 camerapositions.

Stereoptic views produced 412 by the system 10 can be enhanced bycombining images obtained from at least two independent or independentlyoperated cameras 12. In an illustrative system embodiment 10, such asshown in FIG. 1, FIG. 2 and FIG. 3, two independent cameras 12 aremounted on the heads of a helicopter pilot 208 a and a co-pilot 208 b.In some embodiments, the cameras 12 can be enabled for any of nightvision, thermal vision, or other purposes. Because the cameras 12 aremounted to the helmets, e.g., 1002 (FIG. 23), the pilot 208 a andco-pilot 208 b can select what they wish to view, i.e., the scene 212,e.g., 212 a and 212 b, by moving their heads. At any given time, thepilot 208 a and co-pilot 208 b may be looking at different scenes 212,but in many instances there is a significant overlap 214 in the scenes212.

By utilizing position sensors 302 (FIG. 3) mounted in the helmet 1002 orthe camera 12, the system 10 is able to determine 404 (FIG. 4) when thescenes 212 viewed by the pilot 208 a and co-pilot 208 b overlap 214.When such an overlap 214 is determined 404 to exist 410 (FIG. 4), someembodiments of the system can notify 414 (FIG. 4) the pilot 208 a andco-pilot 208 b that an enhanced stereoptic view is available, and/or thesystem 10 can automatically switch 416 (FIG. 4) to provide the enhancedstereoptic view.

This enhanced image is created 412 (FIG. 4) by using the left cameraimage from the individual, e.g., 208 b, seated in the left seat for theleft eye and the right camera image from the individual, e.g., 208 a,seated in the right seat for the right eye. The resulting enhancedstereoptic view has a significantly larger baseline than a human USRwould normally be capable of seeing. This resulting enhanced stereopticview can enable the users USR, e.g., 208 a and/or 208 b, to moreaccurately determine the difference in depth between two or more distantobjects 206 (FIG. 2).

Because the cameras 12 are interdependently operated, the scenes 212,e.g., 212 a, 212 b (FIG. 2) from each individual may not overlapperfectly. To compensate for the lacking scene information, someembodiments of the system 10 can attempt to extrapolate the missinginformation, crop the scenes to only those portions which overlap orsimply leave the missing scene information blacked out, essentiallyresulting in monocular vision in these areas. In some embodiments, theimages are manipulated, such as to compensate for slight variations ofcamera positioning, such as camera height.

Such a system is not limited to the pilot and co-pilot scenario, but maybe utilized in any situation in which there are two or more independentor independently operated cameras 12. This system may also beincorporated in a machine vision system in which otherwise independentcameras 12 can be fused together for specific tasks to allow for betterdepth perception.

For instance, in an illustrative alternate embodiment a pilot 208 a ofan aircraft 202 can have an associated camera 12 and position sensor302, while a second camera 12 can be separated 16 from the first camera,at either a fixed position on the aircraft 202 or movable with respectto the aircraft 202. In some embodiments in which the secondary camera12 is located at a known position, a corresponding position sensor 302is not required. Some embodiments can include more than one fixedcamera, such as opposing fixed secondary cameras located on opposingwings of a fixed wing aircraft 202, in which the secondary camera 12 tobe used may be determined based on any of position, orientation, oravailability. In some embodiments in which the secondary camera 12 ismovable, the position of the camera can be selectively controlled, suchas to aid in any of depth perception or viewing around occlusions orobstacles 206.

An illustrative embodiment can include, inter alias a non-transitorycomputer readable medium having stored thereon a computer program havingmachine-readable instructions for performing, when running on acomputer, a method comprising tracking of the position and orientationof at least two independently operable cameras that are separated fromeach other by a baseline difference, wherein each of the cameras has acorresponding field of vision, determining if the fields of vision ofthe cameras at least partially overlap, using the tracked positions andorientations of the cameras, and producing a stereoscopic view usingimages from the cameras when the fields of vision overlap.

3D Depth Map Used to Augment Stereo Vision

FIG. 5 shows an illustrative system 500 for augmenting stereo visionusing a three-dimensional (3D) depth map 506. FIG. 6 shows anillustrative method 540 for augmenting stereo vision using athree-dimensional (3D) depth map 506.

The illustrative system 500 seen in FIG. 5 includes an image acquisitionand digitization module 502, which is configured to receive images fromone or more imagers 504, such as through a wired or wireless connection524. The illustrative system 500 seen in FIG. 5 also includes a 3D depthmap acquisition module 506, that is configured to receive depth mapinformation for any of sensors, cameras or other imaging devices, suchas through any of wired or wireless connections. One or more modules ofthe illustrative system 500 can be integrated within an enclosure 521,such as a standalone enclosure, or integrated within other devices, suchas within a vehicle or aircraft 202, or within a helmet or goggles,e.g., 20. One or more of the functional modules 502, 506, 514, 516, 518,520 can include any of a local processor 1202 (FIG. 24), or can becontrolled by one or more system processors 1202, such as through asystem bus 1210.

An illustrative binocular color night vision system 500 can use ahigh-resolution depth map 506 to present 568 (FIG. 8) binocular imagesthat are readily fused by the user USR. This provides the user USR withan improved sense of depth and an enhanced understanding of his or hersurroundings.

The system 500 acquires 546 (FIG. 6) a set of images from multipleimagers 504, for example color-filtered image intensifier tubes or imageintensifier tubes with photocathodes with specific spectralsensitivities. Each of the images is then digitized.

The system 500 additionally acquires 548 (FIG. 6) a digital depth map ofthe field of view spanned (in aggregate) one or more of the imagers. Thesystem 500 registers 554 (FIG. 6) each of the acquired images to thedepth map 504.

For each of the acquired images, the system then generates 560 (FIG. 6)a modified image for each of the user's two eyes by applying aperspective transformation, such as based on a human baseline 18 (FIG.1), which in some embodiments corresponds to the distant between displayscreens 22 for goggles, e.g., 20. The perspective transformation shiftseach pixel within the acquired image 502 by an amount determined by (a)the depth of the pixel as indicated by the depth map 504, and (b) theoffset, i.e., the baseline, between the acquiring imager and the user'seye. Each of the modified images 516 thus alters the acquired imagery toappear as it would from one of the user's two eyes.

The resulting pairs of images 516 are digitally overlaid 564 (FIG. 6)and presented 568 (FIG. 6) to the user USR.

The system 500 can acquire 504 the depth map using any of a number oftechniques. For example, the system 500 may use an infrared depth sensor510, e.g., Microsoft Kinect) or a time-of-flight (ToF) camera 512, e.g.,a SwissRanger, such as currently available from Mesa Imaging AG, ofRUschlikon, Switzerland.

In some preferred embodiments 500, the depth map is acquired 504 using aplenoptic camera 508, i.e., a light-field camera, which, as a passiveimager, is advantageous for clandestine applications. In some systemembodiments 500, the plenoptic camera 508 comprises an array ofmicrolenses, with complementary color filters that are assigned toindividual microlenses. For instance, a portion of the micro-lenses ofan imager 504 can be used to acquire 546 the color imagery, while theremaining portion 508 (FIG. 5) of the micro-lenses, which is preferablyinterleaved in some embodiments, can be used to acquire 504 the depthmap.

In some embodiments, any of the acquired images and 3D depth map arefiltered through one or more filter. In some embodiments, the filterscan be cyan, magenta, and yellow, while others may be filtered forinfrared bands and other multispectral information. In some embodimentsthat include microlenses, the microlenses can be left unfiltered. Insome embodiments, the resulting array resembles a Bayer filter, in thatcolor information is gathered while avoiding the need to de-Bayer. Itshould also at the same time generate a depth map 506 for 3Dinformation.

An illustrative embodiment of the binocular night vision method 540comprises, when operating within an image area, acquiring 546 left andright images 502 of the image area, acquiring 548 three-dimensional (3D)depth maps 506 of the image area, registering 554 the acquired images502 with the corresponding 3D depth maps 506, applying 560 perspectivetransformations 516 to match eye positions of a viewer USR, overlaying564 the images to produce binocular images, and presenting 568 thebinocular images to the viewer. In some embodiments, the 3D depth map506 is acquired through any of a plenoptic camera 508, an infrared depthsensor 510, or a time-of-flight camera 512. In some embodiments, thepresented binocular images are configured to provide a viewer USR withany of an improved sense of depth and an enhanced understanding of theviewer's surroundings. In some embodiments the method can includefiltering any of the acquired images 502 and the 3D depth maps 506,wherein the filtering includes any of color filtering though cyan,magenta and yellow filters, infrared filtering, or filtering for othermultispectral information.

Depth Map Applied to Information Overlays To Resolve Occlusions

FIG. 7 is an illustrative view of an image corresponding to a scene 212in which overlaid information 608, 610 can be applied with respect toone or more objects 602, 606. The overlaid information seen in FIG. 7includes a graphic element 608 and/or a label 610, such as to conveyinginformation regarding one or more identified objects 602 and/or 606.

FIGS. 8-10 show schematic views of system embodiments 500 in which adepth map 504 applied to an information overlay to resolve occlusions.

For instance, the illustrative systems 500 can overlay supplementarycontent 608,610 with an appropriate binocular disparity based on thedepth map. For example, informational overlays, i.e., labels,corresponding to objects at particular locations within the field ofview can be generated in each eye with appropriate binocular disparitybased on the value of the depth map 504 at that particular location.

In some embodiments, when the system 500 detects that a tracked object,e.g., 602, has been occluded by another, nearer object, e.g., 606, thesystem can alter the overlay 608, 610, to lessen the potentialdistractions described above, using one of three approaches.

For example, as seen in FIG. 8, the system 500 can remove 640 (FIG. 8)the overlay 608,610 from being displayed, such as to one of the user'seyes. As seen in FIG. 9, some embodiments of the system 500 can render660 the overlay 608,610 in a semi-transparent or otherwise de-emphasizedmanner. As seen in FIG. 10, some embodiments of the system can remove,i.e., “knock out” 680 a portion of the overlay 608,610 that is occludedby a nearer object, e.g., 606.

In the approach 640 seen in FIG. 8, the user USR will not attempt to“fuse” the overlay. The user USR will thus not perceive the label at anyparticular depth, removing any possible conflict with the user's mentalmodel of the scene 212. In the approach 660 seen in FIG. 9, the overlay608,610 can be presented at the depth of either the occluding oroccluded (tracked) object. In the first case 640 seen in FIG. 8, thedepth map 504 is strictly respected. In the second case 660 seen in FIG.9, the overlay is generated at the “last seen” depth of the trackedobject 602. In either case 640,660, the semi-transparent appearance ofthe overlay 608,610 lessens the user's impression of its physicalnature, lessening the impact of the non-physical behavior of either ajump forward to the occluding object depth or a mismatch in occlusionbehavior of the overlay and tracked object. In the third case 680 seenin FIG. 10, the remaining portion of the overlay 608,610 is presented atthe depth of the tracked object 602, removing the non-physical behaviorentirely.

An illustrative embodiment of the method for overlaying information608,610 on an acquired image for presentation to a viewer comprisesacquiring one or more images of a scene at a particular location,wherein the scene includes a plurality of objects, e.g., 602,610, usingone or more image capturing devices each having a corresponding field ofview. The illustrative method tracks an object within the scene, andgenerates an informational overlay 608,610 that corresponds to thetracked object at a particular location within the field of view foreach of the viewer's eyes, with binocular disparity based on a value ofdepth map at the particular location. Upon determining that a trackedobject 602 has been occluded by another object 606 within the acquiredimages, the method alters a display of the informational overlay608,610, based on the occlusion. In some embodiments, the altering ofthe display of the informational overlay 608,610 includes removing theinformational overlay from being displayed to at least one of theviewer's eyes. In some embodiments, the altering the display of theinformational overlay 608,610 includes rendering at least a portion ofthe informational overlay 608,610 in any of a semi-transparent orde-emphasized manner, such as shown in FIG. 9. In some embodiments, thealtering of the display of the informational overlay 608,619 includesremoving a portion of the informational overlay that 608,610 is occludedby the nearer object, such as seen in FIG. 10.

Direct Painting of Overlay Information Onto Phosphor Screen

For some applications, it can be useful to overlay the above symbologyor messages without bulky optical elements or passing through a digitalsensor and display.

FIG. 11 is a schematic view of an illustrative system 700 for directpainting of overlay information onto a phosphor screen 714. FIG. 12 is aflowchart of an illustrative method 740 for direct painting 746 ofoverlay information onto a phosphor screen 714, using the system 700seen in FIG. 11. FIG. 13 is a schematic view of an alternateillustrative system 800 for direct painting of overlay information ontoa phosphor screen 812. FIG. 14 is a flowchart of an alternateillustrative method 840 for direct painting of overlay information ontoa phosphor screen 812, using the system 800 seen in FIG. 13.

To generate symbology on a night vision view, as disclosed herein, someembodiments of the system, e.g., 700,800, can “paint” information withbeams outside the visible spectrum, directly onto a phosphor screen714,812, causing re-emission in the visible spectrum 718,818 to theuser's eye.

In the illustrative embodiment 700 seen in FIG. 11, a beamsplitter 710that is enabled to reflect ultraviolet light and transmit visible lightis positioned 742 between the NVG output 704 and the user's eye. Asteerable laser 706 in the ultraviolet range is aimed 744 at thebeamsplitter 710, such that its beam 708 is reflected 712 towards thephosphor screen 714. When the beam 712 hits the screen 714, its energycauses photons to be emitted 718 towards the user's eye. Thus, asteerable UV beam 708 can paint symbology, e.g., 608, 610 (FIGS. 7-10)onto the phosphor screen 714, which the user will see overlaid 746 onthe amplified analog image 704.

In an illustrative embodiment a method comprises positioning abeamsplitter 710 that is enabled to reflect light 708 outside thevisible spectrum, and to transmit visible light 704 between the outputof an image intensifier 702 associated with night vision goggles (NVG)and a phosphor screen 714. The illustrative method aims a steerablelaser 706 having an output beam 708 of the light outside the visiblespectrum at the beam splitter 710, such that the light 708 outside thevisible spectrum is reflected toward the phosphor screen 714, whereinthe light outside the visible spectrum includes information, e.g.,608,610, wherein the output of the beamsplitter 710 includes both thevisible output 704 of an image intensifier and the light 708 outside thevisible spectrum that includes the information, and wherein the outputof the beamsplitter 710 is painted 713 directly on the phosphor screen714, to be emitted 718 from the phosphor screen in a visible spectrumfor viewing by a viewer USR. In some embodiments, the light 708 outsidethe visible spectrum is ultraviolet (UV) light.

In the second embodiment 800 seen in FIG. 13, the output 804 of a verylow-powered infrared laser 802 is aimed 842 at the photocathode 808 atthe collection end 807 of the NVG intensifier 806. The IR energy 804impacts the visible and IR sensitive photocathode 808, and causeselectrons 810 to emit towards the microchannel plate 812 to be amplified844. In a similar manner to the embodiment 700, the steerable IR laser802 can be enabled to draw information 804 onto the photocathode 808,wherein the information 804 is amplified 844, along with the scene 212and transmitted 846 to the eye of the user USR.

An illustrative method comprises aiming an infrared (IR) laser 804having an IR output beam 804 at a photocathode 808 located at acollection end 807 of an image intensifier 806 associated with nightvision goggles (NVG), wherein the photocathode 808 is sensitive tovisible energy from a received image signal 803 and IR energy. Theillustrative method directs 810 the combined visible energy 803 and IRenergy 804 toward a microchannel plate 812, to be amplified as acombined visible output signal 818, which can be directed for viewing bya user USR. In some embodiments, the IR output beam includesinformation, e.g., text, symbols, and/or other graphics, wherein theamplified combined visible output signal 818 includes the information.

Generating Stereo Information from a Single Lens

FIG. 15 is an illustrative view of a system 860 for generating stereoinformation from a single lens 864. FIG. 16 is a flowchart of anillustrative method 880 for generating stereo information from a singlelens 864.

In the illustrative system 860 seen in FIG. 15, a circular mask 868 isplaced behind a lens 864, wherein the circular mask 868 includes threeapertures 869 defined therethrough, containing a red filter 870 r, ablue filter 870 b, and a green filter 870 g. Incident light 862 isreceived 882 through the lens 864 and is directed 866 through thefilters 870, e.g., 870 r, 870 b, 870 g. A Bayer-pattern sensor 874 isconfigured to receive 884 light 872 from each of the three apertures869, wherein each pixel in the Bayer-pattern sensor 874 only acceptslight of its color. The output of the Bayer-pattern sensor 874 can beused to generate 886 stereo information on a limited baseline, such ascorresponding to the up to the width of the lens (e.g., 15 mm).

An illustrative method comprises receiving light 866, that istransmitted 862 through a single lens 864, at a mask 868, e.g., acircular mask 868 that includes three apertures 869 definedtherethrough, containing a red filter 870 r, a blue filter 870 b, and agreen filter 870 g, wherein incident light 862 that is received throughthe lens 864 is directed through the filters 870. The method thenreceives the light 872 directed from each the filters 870 with aBayer-pattern sensor 874 having associated pixels, wherein each pixel inthe Bayer-pattern sensor 874 only accepts light 872 of a correspondingcolor. The method generates stereo information with the output of theBayer-pattern sensor 874, wherein the stereo information corresponds toa baseline, which can have a distance that is less than or equal to thewidth of the lens.

FIG. 17 is a schematic view of an alternate illustrative system 900 forgenerating stereo information from a single lens 906. FIG. 18 is aflowchart of an illustrative method 940 for generating stereoinformation from a single lens 960, such as using the system 900 seen inFIG. 17.

Some embodiments of the alternate illustrative system 900 seen in FIG.17 can be used for remote controlled vehicles 902, which are sometimesused to explore environments 920, e.g., underground passages 920, andcan be enabled to avoid detection. As a result, some of these vehicles902 are equipped with night vision sensors 906, which allow an aboveground remote operator RO to control and navigate the vehicle 902 with aremote device or system 910. Night vision sensors 906, specificallycolor night vision sensors, can be expensive, large and sensitive tomovement. In contrast to prior systems, some systems are enabled to usemonocular vision, which results in very poor depth perception, such asthe inability of the operator to determine the scale of vertical objectssuch as holes on the ground.

In the embodiment 900 shown in FIG. 17, a sensor 906 is oriented so thatit looks forward from the vehicle 902 on a plane parallel to the ground.A periscope like device 904 is then placed inline with the night visionsystem 900. The periscope 904 is affixed to a pivot 905 on the verticalplane along the z-axis of the night vision system. As a result, in someembodiments, the periscope 904 can rotate 908 so that it provides a viewfrom both a higher and lower perspective. In some embodiments, theperiscope 904 can also be rotated to provide a right and leftperspective.

In some embodiments, the periscope 904 constantly rotates, e.g., at arate of at least 30 times per second. During this rotation, the nightvision system 900 can constantly record images. The video captured bythe night vision system 900 can be transmitted 912 back to the remoteoperator RO. If this feed were to be viewed without further processing,it would display the view directly in front of the remote vehicle 902.However, the perspective of this view would be constantly shifting atleast 30 times per second.

Instead, before the resulting video feed is displayed to the operatorRO, some embodiments of the system 900 can select two staticperspectives, which represent the stereo pairs for binocular vision. Thesystem then displays only video captured at these two locations to theoperator RO, such as through the remote device and/or though associatedgoggles, e.g., 20 (FIG. 1). In some embodiments, some or all of thevideo from the other perspectives is discarded.

The video feed described above can be displayed to the operator RO as aleft and right video feed, likely through a binocular viewing system,such as an Oculus Rift virtual reality headset, available through OculusVR, LLC, Irvine, Calif. In doing so, the system provides the operator ROwith a binocular view of the underground location 920 in a manner inwhich the operator RO is able to perceive a sense of depth.

In a current default embodiment, the system 900 selects the two staticperspectives as the right-most and left-most perspectives correspondingto 3 and 9 o-clock on a clock face. This orientation provides thelargest stereo baseline, which can allow the operator RO to perceivedepth even in very distant objects.

In some embodiments, the operator RO can select various stereo pairs.For example, the operator RO may select the 1 and 11 o-clock positions,in which this view would provide the operator RO with a narrow baselinethat is appropriate for viewing objects up close. In addition, this viewcan give the operator RO the perception of having “popped his head up”,because the perspective is now more elevated than the defaultperspective. This elevated perspective can be useful in discerning thescale of vertical objects, such as holes in the ground.

In some embodiments, the remote operator RO can control these variousperspectives by simply raising and lowering his or her head if thesystem is equipped with head tracking. Alternatively, the system 900 mayemploy a simple joystick that enables the operator to raise and lowerhis or her perspective.

An illustrative method comprises orienting a night vision sensor 906 ona periscope mount 904 so that the night vision sensor 906 looks forwardfrom a remotely controlled vehicle on a plane, e.g., 220 x, parallel toa ground surface 920, wherein the periscope mount 904 is affixed to apivot 905 on a vertical plane, e.g., 220 z, that extends perpendicularlyto the ground surface 920. The illustrative method controllably rotatesthe periscope mount 904 on the pivot 205 to provide perspective views,while capturing video images, and transmits 912 the video images to aremote device 910, such as corresponding to a remote operator RO. Insome embodiments, the perspective views are any of higher and lowerperspectives, or right and left perspectives. In some embodiments, theperiscope mount 905 is constantly rotated at a frequency that matches aframe rate of a display device, e.g., 910. In some embodiments, theperspective views are selectable by the remote operator. In someembodiments, the remote operator RO can view the images as binocularimages with a binocular viewing system, e.g., display goggles.

Stereo Color Image from a Single Color Channel

Current color night vision methods require a degree of complexity, suchas separate intensifier tubes for each color or spinning filter discs. Amethod of reducing complexity and cost would increase acceptance of CNV.

FIG. 19 is a schematic view of an illustrative night vision device 960that is configured for binocular display of the output of differentoptical devices, e.g., 962,966. In an illustrative color night visiondevice 960, one eye views a high resolution monochrome (white) image 964from a single intensifier tube 962, while the other eye views a lowerresolution color image 968, from a second optical device 966, such asincluding either a second intensifier, or an LCD display fusing theoutput from multiple intensifiers. It has been demonstrated that thehuman brain can combined these inputs 964,968 into a high resolutioncolor image.

Although the monochromatic image 984 seen in FIG. 19 can be alternativecolors, a white image works well. In one embodiment, the intensifier 982is an off-the-shelf white phosphor monocular unit, while the color imageis supplied by a Sony a7S 12 Mega Pixel (MP) digital camera, fitted witha zoom lens to match the field of view of the intensifier 982.

An illustrative night vision device 960 comprises a binocular viewingdevice 20 for viewing by a user, wherein the binocular viewing device 20includes a first display 22 and a second display 22, a first opticaldevice including an intensifier tube 962 configured to provide amonochrome image 964 to the first display 22, wherein the monochromeimage 964 has a first resolution, and a second optical device 966configured to provide a color image 968 to the second display 22,wherein the color image 968 has a second resolution, wherein the secondresolution is lower than the first resolution of the monochrome image964. In some embodiments, the monochrome image 964 and the color image968 are configured to be displayed separately and simultaneously to theuser through the displays 22, and may be perceived as a combined highresolution color image by the user USR.

FIG. 20 is a schematic view of an illustrative system 980 for generatinga composited image signal 992 from different optical devices 982,986,using a variable beamsplitter. For instance, a highly sensitive colorcamera 986 can be used to acquire a color image 988 down to a minimumillumination level, while a traditional image intensifier 982 gathersmonochromatic imagery 984 at lower illumination levels. Low-lighttechnology has taken a dramatic leap forward recently, such as with theimplementation of a 4 million ISO sensor in the Canon ME20E-SH camera.Using large-scale pixels, this sensor is capable of HD video down to0.0005 lux, or a moonless night. This opens up possibilities in usinghigh-ISO visible light cameras as a partial solution to night vision.

The two sources 982,986 are composited within an optical train using avariable beamsplitter 990, whereby the user can controllably vary theratio between the two image sources 982,986. At sufficiently highillumination levels, such as a half-moon, the imagery 992 couldpredominately be derived from the color camera 986, while on overcastnights the intensifier 982 would be favored.

The beamsplitter 990 seen in FIG. 20 may comprise a rotary wheel onwhich transmission-reflection split varies with the angular positionaround the wheel. The user can adjust the relative fraction contributedby each source 982,986, or A/B toggle entirely between one source andthe other.

An illustrative device 980 comprises a variable beamsplitter 990, acolor camera 986 configured to acquire a color image down to a minimumillumination level, and an image intensifier 982 configured to acquire amonochromatic image at a low illumination level, wherein the variablebeamsplitter 990 is configured to receive outputs 988, 984 from thecolor camera 986 and from the image intensifier 982, wherein the outputs988,984 of the color camera 986 and the image intensifier 982 arecombinable 992 in a ratio that is selectable by a user USR, such as toproduce a combined output signal 992, which can be displayed, e.g.,through display goggles 20.

3D Focus Techniques for Head-Mounted Night Vision Goggles

FIG. 21 is a schematic view 1000 of an illustrative autofocus system1005 for head-mounted night vision goggles 1020, which is configured toset the focus of the goggles 1020 for near vision when the user's headHD is tilted down. FIG. 22 is a schematic view of an illustrativeautofocus system for head-mounted night vision goggles, which isconfigured to set the focus of the goggles 1020 for far vision when theuser's head HD is looking at the horizon or vertically upward.

As part of an autofocus system for the 3D or 2D systems disclosedherein, a tilt sensor 1022 can be incorporated into head-mounted nightvision goggles 1020,. When the user USR tilts his or her head HD down,e.g., with an angle 1012 below horizontal 220 s, some system embodiments1005 assume that the user USR is looking at the ground 1014 or atanother nearby object, and correspondingly the focus is set for near. Insome embodiments, when the tilt sensor 1022 indicates that the goggles20 are level 220 x or looking upward, the focus is set to a far distanceor infinity, on the assumption the user USR is looking up at the sky orat the horizon.

Additionally, in some embodiments, the NVG lenses, e.g., 22 (FIG. 1) canbe set to converge in steps, to simulate distance. For instance, threelevels of focus can be implemented in an illustrative embodiment: closeor map focus; middle or instrument focus; and far focus.

An illustrative autofocus system for a two dimensional (2D) or threedimensional (3D) display system to be worn by a user USR comprises asensor configured for determining a tilt angle of the display system,and a processor configured to adjust the focus of the display systembased on the determined tile angle. In some embodiments, the processoris configured to set the focus of the display device for near visionwhen the display device is tilted downward. In some embodiments, the isconfigured to set the focus of the display device for far vision whenthe display device is tilted at the horizon or vertically upward. Insome embodiments, the processor is configured to provide any of a closeor map focus, a middle or instrument focus, and a far focus.

An alternate method for adjusting focus of NVGs can be accomplished by asensor 1022 detecting movement of an eyebrow EB of the user USR, or bythe user USR puffing a breath of air with their mouth MO upwards onto asensor 1022.

Synchronized Pulsed IR Flash to Blind Adversary Night Vision

FIG. 23 is a schematic view 1100 of illustrative optical devices 1104and 1120 for offensive and defensive operations.

The well-known phenomenon of night vision blinding can be usedoffensively, that is by causing a bright flash 1106 to disable anenemy's night vision. Intensifiers, e.g., 702 (FIG. 11) have a naturaltemporal frequency response to external light, based on driveelectronics and phosphor physics. As seen in FIG. 23, an intense light1106 may be pulsed at an adversary or enemy with sufficient frequencysuch that when their systems start to recover from the pulse, they arehit again with another pulse in time to prevent recovery, keeping theadversary system blind.

As seen in FIG. 23, an offensive embodiment 1104 can be configured totemporarily blind adversaries, using night vision devices (NVDs). Theembodiment includes a bright IR emitter strobe light 1104 that in someembodiments can be mounted on a soldier's helmet 1102, or included aspart of a “light grenade”. In some embodiments, the frequency of the IRlight 1106 can be selected to be one that the enemy night vision devices(NVDs) 1120 are particularly susceptible to.

To prevent blinding friendly forces, the NVDs 1120 of friendly forcescan be equipped with a number of different features. In one form, anarrow frequency band filter can be used which blocks the IR flash fromthe strobe, but not other IR frequencies. Alternatively, someembodiments of the NVDs 1120 can be equipped with a gating feature thatdisables the NVDs 1120 for very short periods of time, on the order ofmilliseconds. This gating would be specifically timed to coincide withthe strobing of the IR light, and would in effect be a notch filtercoordinated with the pulsed light source.

An illustrative device for enhanced night vision in an environment thatincludes a strobed IR light signal comprises night vision goggles 1120for use by the user USR, and a mechanism 1122 to compensate for thestrobed IR light 1106, wherein the mechanism 1122 includes any of afilter that is configured to block the strobed IR light signal 1106, butallow passage of other IR frequencies, or a gating feature that disablesthe night vision goggles 1120 for short periods of time, wherein thegating is timed to coincide with the arrival of the strobed IR lightsignal 1106.

FIG. 24 is a high-level block diagram showing an example of a processingdevice 1200 that can represent any of the systems described above. Anyof these systems may include two or more processing devices such asrepresented in FIG. 22, which may be coupled to each other via a networkor multiple networks.

In the illustrated embodiment, the processing system 1200 includes oneor more processors 1202, memory 1204, a communication device 1206, andone or more input/output (I/O) devices 1208, all coupled to each otherthrough an interconnect 1210. The interconnect 1210 may be or includeone or more conductive traces, buses, point-to-point connections,controllers, adapters and/or other conventional connection devices. Theprocessor(s) 1202 may be or include, for example, one or moregeneral-purpose programmable microprocessors, microcontrollers,application specific integrated circuits (ASICs), programmable gatearrays, or the like, or a combination of such devices. The processor(s)1002 control the overall operation of the processing device 1200. Memory1004 may be or include one or more physical storage devices, which maybe in the form of random access memory (RAM), read-only memory (ROM)(which may be erasable and programmable), flash memory, miniature harddisk drive, or other suitable type of storage device, or a combinationof such devices. Memory 1204 may store data and instructions thatconfigure the processor(s) 1202 to execute operations in accordance withthe techniques described above. The communication device 1206 may be orinclude, for example, an Ethernet adapter, cable modem, Wi-Fi adapter,cellular transceiver, Bluetooth transceiver, or the like, or acombination thereof. Depending on the specific nature and purpose of theprocessing device 1200, the I/O devices 1208 can include devices such asa display (which may be a touch screen display), audio speaker,keyboard, mouse or other pointing device, microphone, camera, etc.

Unless contrary to physical possibility, it is envisioned that (i) themethods/steps described above may be performed in any sequence and/or inany combination, and that (ii) the components of respective embodimentsmay be combined in any manner.

Some of techniques introduced above can be implemented by usingprogrammable circuitry programmed/configured by software and/orfirmware, or entirely by special-purpose circuitry, or by a combinationof such forms. Such special-purpose circuitry (if any) can be in theform of, for example, one or more application-specific integratedcircuits (ASICs), programmable logic devices (PLDs), field-programmablegate arrays (FPGAs), etc.

Software or firmware to implement the techniques introduced here may bestored on a machine-readable storage medium, e.g., a non-transitorycomputer readable medium. and may be executed by one or moregeneral-purpose or special-purpose programmable microprocessors. A“machine-readable medium”, as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media, e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.

Note that any and all of the embodiments described above can be combinedwith each other, except to the extent that it may be stated otherwiseabove or to the extent that any such embodiments might be mutuallyexclusive in function and/or structure.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the examplesdisclosed herein. Accordingly, the specification and drawings are to beregarded in an illustrative sense rather than a restrictive sense.

What is claimed is:
 1. A processor implemented method, comprising thesteps of: tracking, via the processor, one or more positions,orientations, and fields of vision of a plurality of independent imagingdevices, wherein each of the plurality of independent imaging devices ismovably fixed along a baseline of adjacent positions; determining, bythe processor, an overlap between the fields of vision of a firstimaging device and a second imaging device of the plurality ofindependent imaging devices, based on a tracked position and orientationof each of the first and second imaging devices; capturing a first imagefrom the first imaging device and capturing a second image from thesecond imaging device, via the processor; generating, by the processor,a stereoscopic view by using the first and second captured images toproduce the stereoscopic view, when the fields of vision of the firstand second imaging devices at least partially overlap; and displayingthe stereoscopic view to a display system, via the processor, whereinthe display system outputs stereoscopic view to a first screen portionand a second screen portion, oriented respectively within the displaysystem.
 2. The method of claim 1, further comprising the steps of:receiving, at the processor, a designation of one or more positionsalong the baseline of adjacent positions; and capturing images, via theprocessor, from one or more imaging devices of the plurality ofindependent imaging devices, which are currently fixed to the one ormore designated positions respectively, for the generating of thestereoscopic view.
 3. The method of claim 1, further comprising thesteps of: receiving, at the processor, a designation of one or morepositions along the baseline of adjacent positions; moving, via theprocessor, one or more imaging devices of the plurality of independentimaging devices, to the one or more designated positions; and capturingimages, via the processor, from the one or more moved imaging devices,for the generating of the stereoscopic view.
 4. The method of claim 2,wherein the first imaging device is fixed at a predefined position andthe method steps further comprise the steps of: capturing, via theprocessor, the first image from the first imaging device; capturing, viathe processor, the second image from the second imaging device, whereinthe second imaging device is located at one of the designated positions;and generating, by the processor, the stereoscopic view, when the fieldsof vision of the first and second imaging devices at least partiallyoverlap.
 5. The method of claim 3, wherein the first imaging device isfixed at a predefined position and the method steps further comprise thesteps of: capturing, via the processor, the first image from the firstimaging device; capturing, via the processor, the second image from thesecond imaging device, wherein the second imaging device is located atone of the designated positions; and generating, by the processor, thestereoscopic view, when the fields of vision of the first and secondimaging devices at least partially overlap.
 6. The method of claim 1,further comprising the steps of: receiving, at the processor, adesignation of two positions along the baseline of adjacent positions;moving, via the processor, two imaging devices of the plurality ofindependent imaging devices, to the two designated positions,respectively; capturing images, via the processor, from the two movedimaging devices, respectively; and generating, by the processor, thestereoscopic view, when the fields of vision of the two imaging devicesat least partially overlap.
 7. The method of claim 1, wherein eachimaging device of the plurality of independent imaging devices includesa leftward oriented imager and a rightward oriented imager, orientedwith respect to each other, within each imaging device, and the step ofcapturing a first image from the first imaging device and capturing asecond image from the second imaging device further comprises: capturingsaid first image from any of the oriented imagers of the first imagingdevice, while simultaneously capturing said second image from any of theoriented imagers of the second imaging device.
 8. The method of claim 7,wherein the step of displaying the stereoscopic view to a display systemfurther comprises: displaying, via the processor, said first image tothe first screen portion of the display system, and simultaneouslydisplaying, via the processor, said second image to the second screenportion of the display system, when fields of vision of one or more ofthe oriented imagers of the first imaging device, and fields of visionof one or more of the oriented imagers of the second imaging device, atleast partially overlap.
 9. The method of claim 1, further comprisingthe steps of: receiving, at the processor, a specification of a baselinedistance; and controlling, via the processor, one or more imagingdevices of the plurality of independent imaging devices, to repositionand/or select one or more of the imaging devices along the baseline ofadjacent positions, based on the received specified baseline distance.10. The method of claim 9, wherein the first imaging device is fixed ata predefined position, the method further comprises the step of:repositioning and/or selecting one of the one or more independentimaging devices which has a separation distance approximate to that ofthe specified baseline distance with respect to the first imagingdevice, wherein the separation distance is a distance between the firstimaging device and the selected imaging device.
 11. The method of claim1, wherein the first imaging device and the second imaging device detectand output infrared, time of flight, and/or light field data as images.12. The method of claim 8, wherein the step of displaying thestereoscopic view to a display system further comprises: displaying, viathe processor, the stereoscopic view, to a first and second screenportion of a head-mounted-display device.
 13. A vision system,comprising: a plurality of independent imaging devices positioned alonga baseline of distinct adjacent positions, each with a correspondingpositions sensor, orientation sensor, and an associated field of vision;a display system including a first screen portion and a second screenportion, oriented respectively; and a processor that is configured to:track, one or more positions, orientations, and fields of vision of theplurality of independent imaging devices, wherein each of the pluralityof independent imaging devices is movably fixable along the baseline ofdistinct adjacent positions; determine a pair of imaging devices fromthe plurality of independent imaging devices with overlapping fields ofvision, based on the tracked position and orientation of each of theimaging devices; capture a first image from a first imaging device withan overlapping field of vision with a second imaging device of theplurality of independent imaging devices, and capture a second imagefrom the second imaging device which has an overlapping field of visionwith the first imaging device; generate a stereoscopic view by using thefirst and second captured images to produce the stereoscopic view, whenthe fields of vision of the first and second imaging devices at leastpartially overlap; and display the stereoscopic view to the displaysystem, wherein the display system outputs the stereoscopic view to thefirst screen portion and the second screen portion.
 14. The visionsystem of claim 13, wherein the display system further comprises one ormore head-mounted display devices.
 15. The vision system of claim 14,wherein the head-mounted display devices include any of a helmet,goggles or glasses, wherein the first screen portion is a right displayscreen and the second screen portion is a left display screen configuredto display a corresponding right image and left image respectively. 16.The vision system of claim 13, wherein each of the plurality ofindependent imaging devices include a right imager and a left imager, inrelation to each other, that are independently operable.
 17. The visionsystem of claim 13, wherein each of the plurality of independent imagingdevices include one or more of a time of flight imager, an infraredimager, a plenoptic imager, a thermal imager, and an optical imager. 18.The vision system of claim 13, wherein the processor is furtherconfigured to: receive a designation of one or more positions along thebaseline of distinct adjacent positions; and reposition one or moreimaging devices of the plurality of independent imaging devices, to theone or more designated positions.
 19. A non-transitory computer readablemedium having stored thereon a computer program having machine-readableinstructions for performing, when executed by one or more processors,steps comprising: tracking one or more positions, orientations, andfields of vision of a plurality of independent imaging devices, whereineach of the plurality of independent imaging devices is movably fixedalong a baseline of distinct adjacent positions; determining an overlapbetween the fields of vision of a first imaging device and a secondimaging device of the plurality of independent imaging devices, based ona tracked position and orientation of each of the first and secondimaging devices; capturing a first image from the first imaging deviceand capturing a second image from the second imaging device,simultaneously; generating a stereoscopic view by using the first andsecond captured images to produce the stereoscopic view, when the fieldsof vision of the first and second imaging devices at least partiallyoverlap; and displaying the stereoscopic view to a display system,wherein the display system outputs the stereoscopic view to a firstscreen portion and a second screen portion, oriented respectively withinthe display system.
 20. The non-transitory computer readable medium ofclaim 19, further comprising machine-readable instructions forperforming, when executed by one or more processors, steps comprising:receiving a designation of one or more positions along the baseline ofdistinct adjacent positions; and repositioning one or more imagingdevices of the plurality of independent imaging devices, to the one ormore designated positions.